Papers with action prediction
Multitask Multimodal Prompted Training for Interactive Embodied Task Completion (2023.emnlp-main)
Copied to clipboard
Georgios Pantazopoulos, Malvina Nikandrou, Amit Parekh, Bhathiya Hemanthage, Arash Eshghi, Ioannis Konstas, Verena Rieser, Oliver Lemon, Alessandro Suglia
| Challenge: | Embodied MultiModal Agent (EMMA) is a unified encoder-decoder model that reasons over images and trajectories and casts action prediction as multimodal text generation. |
| Approach: | They propose an Embodied MultiModal Agent (EMMA) that uses a unified encoder-decoder model that reasons over images and trajectories and casts action prediction as multimodal text. |
| Outcome: | The proposed model performs on par with similar models on several VL benchmarks and sets a new state-of-the-art success rate on the Dialog-guided Task Completion (DTC) benchmark. |
Dynamic Planning for LLM-based Graphical User Interface Automation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to planning for GUI tasks are limited due to long historical dialogues. |
| Approach: | They propose a novel approach to dynamic planning based on environmental feedback and execution history to guide action prediction in GUI tasks. |
| Outcome: | The proposed approach surpasses the strong GPT-4V baseline by +12.7% in accuracy. |
Harry Potter and the Action Prediction Challenge from Natural Language (N19-1)
Copied to clipboard
| Challenge: | Using textual descriptions of scenes, we explore the challenge of action prediction from textual description. |
| Approach: | They propose a testbed to approximate whether text inference can be used to predict upcoming actions from textual descriptions of scenes. |
| Outcome: | The proposed model performs best for frequent actions and large scene descriptions, but logistic regression fails on infrequent actions. |
Commonsense Justification for Action Explanation (D18-1)
Copied to clipboard
| Challenge: | a recent study examines the commonsense reasoning used by humans to justify an AI prediction. |
| Approach: | They propose an approach that models object relations/attributes of the world as latent variables and jointly learns a performer that predicts actions and an explainer that gathers commonsense evidence to justify the action. |
| Outcome: | The proposed model achieves significantly higher performance in both action prediction and justification. |
Are Rules Meant to be Broken? Understanding Multilingual Moral Reasoning as a Computational Pipeline with UniMoral (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to analyze moral reasoning are discordant and lack cohesion, focusing on isolated aspects of the process. |
| Approach: | They propose a unified dataset that integrates moral dilemmas annotated with labels for action choices, ethical principles, contributing factors, and consequences, and captures diverse socio-cultural contexts. |
| Outcome: | The proposed dataset integrates moral dilemmas annotated with labels for action choices, ethical principles, contributing factors, and consequences, along with annotators’ moral and cultural profiles. |
CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation (2024.findings-acl)
Copied to clipboard
| Challenge: | Current vital challenges for autonomous agents lie in two aspects: dependence on strong (M)LLMs and insufficient GUI environment modeling. |
| Approach: | They propose a comprehensive cognitive LLM agent with two novel approaches to improve GUI automation performance. |
| Outcome: | The proposed agent achieves state-of-the-art performance on AITW and META-GUI benchmarks. |
Android in the Zoo: Chain-of-Action-Thought for GUI Agents (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on large language models (LLMs) focus on the semantics of smartphone operations. |
| Approach: | They propose a large language model (LLM) which predicts a sequence of actions of API by analyzing past actions and visual observations. |
| Outcome: | The proposed model improves the prediction of actions on a zero-shot Android-In-The-Zoo dataset compared to previous models . |